SUBSTITUTE SPECIFICATION 

TITLE OF THE INVENTION 

METHOD AND DEVICE FOR INPUTTING CONTROL INFORMATION IN COMPUTER 

SYSTEMS 

[0001 J This application is based on and hereby claims priority to POT Application No. 
PCT/EPOO/05917 filed on June 26, 2000, and European Application No. 99112287.0 filed June 
25, 1999 in Europe, the contents of which are hereby incorporated by reference. 

BACKGROUND OF THE INVENTION 

[0002] The invention relates to a method and a device for inputting control information in 
computer systems, and in particular to an input device and a pertaining method for operating a 
mobile multimedia communication terminal (mobile videophone). 

[0003] Mobile communication terminals such as mobile phones for example increasingly have 
access to "online services". In particular the introduction of the future broadband UMTS mobile 
communication network will enable high-quality mobile Internet access. 

[0004] Likewise, ultra-small computer systems (such as palmtop computers for example) on 
which a wide variety of organizer functions (such as address management, diary, task list, etc. 
for example) are implemented are becoming increasingly important. 

[0005] The operation of these display-driven computer systems requires an input device with 
which the information displayed on a display unit can be controlled or selected. 

[0006] Mobile ultra-small computer systems of this type usually have so-called "touch 
screens" as an input device, wherein the display unit is equipped with a touch-sensitive layer 
and it is accordingly possible either to input control information or select the displayed 
information by an additional input stylus. The disadvantage of such input devices however is 
that the additional input stylus has to be accommodated in the device and consequently may get 
lost. Moreover, user interfaces for these "touch-screen" display units are mechanically not very 
robust. 

[0007] So-called "mouse" input devices are also known, wherein a rolling ball is converted into 
a corresponding electrical signal which is displayed as a pointer symbol or selection information 
on a display unit. In addition, control keys can be used as an input device for computer 
systems. 
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[0008] Particularly in the case of ultra-small computer systems however, input devices of this 
type are not possible because of the greater amount of space required. 

[0009] As an alternative to the touch-controlled input devices described above, so-called non- 
contact input devices are known in which, for example, a video camera records a position of an 
input stylus, evaluates it. and on the basis of the evaluation a pointer symbol or selection 
information displayed on the display unit is subsequently displaced (e.g. US 5.726,685). 
Furthemiore. a so-called gesture input device is known from the publication US 5.617,312 in 
which the gestures of a person are recorded by a camera and are subsequently converted into 
corresponding control signals for controlling a pointer symbol or selection Infomnation displayed 
on the display unit. However, since said input devices are based on relatively complex pattern 
recognition, they are usually prone to errors and require large amounts of computing power. 

[0010] Known from the publication EP-A-773 494 is an input device for computer systems 
having a recording unit for serially recording image infomnation; an image evaluation unit for 
evaluating the recorded image information and determining control infomnation; a data 
processing unit for processing the control information in selection Infomnation; and a display unit 
for displaying the selection infomnation. The image evaluation unit determines the control 
information from a relative displacement of the serially recorded image information. 

[0011] A device and a method for controlling a display is also known from publication 

JP 10 240 436 A, in which a picture or menu displayed on the display is displaced vertically or 

horizontally respectively depending on a relative movement or displacement of a CCD camera. 

SUMMARY OF THE INVENTION 

[0012] In contrast, one possible object of the invention is to create an input device and a 
pertaining method for computer systems which is space-saving, cost-effective and reliable. 

[0013] Particularly on mobile videophones, the use of an image evaluation unit in particular, 
which uses an image compression device for generating motion vectors, with the control 
information being derived from a main motion vector, enables the recording unit already present 
as well as the associated image compression method to be used simultaneously as an input 
device. 

[0014] Respective parallel displacement components of the image information in the x and y 
directions are preferably used to generate the x and y components of the control information, 
which enables control of a pointer symbol or selection information to be realized on a two- 
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dimensional display. The control information may however also have a z component which is 
produced from a concentric displacement of the image information in the x and y directions, 
which enables control of a pointer symbol or selection information to be realized on a three- 
dimensional or quasi three-dimensional display. In addition, this enables an intuitive 
enlargement or reduction of the display scale. To produce such a concentric displacement of 
image information for three-dimensional input, the recording unit can preferably have a zoom 
function unit. 

[0015] A mobile multimedia communication terminal is preferably used as the computer system, 
with the recording unit in a mobile videophone being used both for recording a call partner and 
as an input device. Analogously, a display unit is used both for displaying a call partner and for 
displaying an Internet page, organizer page, etc. An extremely cost-effective and space-saving 
realization of the input device is thus achieved. The recording unit comprises in particular a 
CCD or CMOS sensor camera, which reduces the space requirement still further. 

[0016] The recording unit preferably records image infomiation for input control in the display 
direction of the display unit, but it can also record image infomiation in the opposite direction. 
This is particulariy advantageous when the input device is provided in a mobile videophone, 
because when the videophone is placed on a suitable surface, by moving the videophone to 
and fro, an associated pointer symbol can be con-espondingly displaced on the display unit in 
the same way as with a mouse input device. 

[0017] The recording unit preferably has a macro camera system for recording image 
information from a very close distance, which enables, for example, an input to be made by 
placing a finger directly on the recording unit. If the recording unit additionally has a pressure 
sensor for confirming displayed selection information, then the input device can be operated in 
the same way as a touch-screen input device. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0018] These and other objects and advantages of the present invention will become more 
apparent and more readily appreciated from the following description of the preferred 
embodiments, taken in conjunction with the accompanying drawings of which: 

Fig. 1 shows a schematic diagram of the input device according to one aspect of the 
invention; 
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Fig. 2 shows a schematic diagram of serially recorded image information to illustrate 
three-dimensional control; 

Figs. 3a and 3b show schematic diagrams of a mobile communication terminal having 
an input device according to a first exemplary embodiment; 

Fig. 4 shows a schematic diagram^of a mobile communication terminal having an input 
device according to a second exemplary embodiment; and 

Figs. 5a to 5c show schematic diagrams of parts of a mobile communication terminal 
having an input device according to a third exemplary embodiment. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

[0019] Reference will now be made in detail to the prefen^ed embodiments of the present 
invention, examples of which are illustrated in the accompanying drawings, wherein like 
reference numerals refer to like elements throughout. 

[0020] Fig. 1 shows a schematic diagram of an input device. In Fig. 1 , reference numeral 1 
denotes a recording unit for serially recording image information, wherein first image information 
A essentially comprises a landscape with a house, two trees and an industrial building. After 
swiveling the recording unit 1 in the negative y direction, second image infonnation A' is 
recorded, which essentially comprises the house, the tree and the industrial building of the first 
image information A plus an additional fence. Owing to the swiveling of the recording unit 1 , the 
additional tree present in the first image information A is not present in the second image 
information A'. 

[0021] An image evaluation unit 3 (AW) serves to evaluate the serially recorded first and 
second image infonnation A and A', with control infomriation x, y and z being determined. 
Evaluation of the image evaluation unit 3 is essentially based here on determining a relative 
displacement between the serially recorded image infonnation, ie. between the first image 
information A and the second image information A". To be more precise, a main displacement 
vector HV which con^esponds to a displacement of the main image components within image 
information A and A' is determined. 

[0022] Given the swiveling of the recording unit 1 in the y direction illustrated in Fig. 1, when 
the image information A and A' is evaluated in the evaluation unit 3. the main displacement 
vector HV is accordingly determined, which essentially represents a displacement in the y 
direction. The control information in the y direction thus obtained is subsequently processed in 
a data processing unit 4 (DV) In such a way that a display unit 2 connected to the data 
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processing unit 4 causes a corresponding displacement of selection infomnation P displayed on 
the display unit 2 to P. In Fig. 1 , the selection information displayed on the display unit 2 is a 
pointer symbol P, which is located in an upper position for a first camera position for recording 
the first image information A, and moves to a lower position (P*) when the recording unit 1 is 
swiveled to record the second image information A*. 

[0023] In accordance with Fig. 1 , therefore, a relative displacement of serially recorded image 
infomnation can be produced and evaluated by swiveling a recording unit 1, which infonnation 
produces a displacement of the displayed pointer symbols P to P' on the display unit 2. 

[0024] The relative displacement is detemiined here on the basis of the principal structures of 
image information A and A\ To be more precise, the relative displacement is not determined on 
the basis of micro displacements, but rather on the basis of macro displacements in the image 
information A and A'. Macro displacements are understood to be, for example, displacements 
of the general background of the image information A and A', while micro displacements are 
displacements of limited image areas as a result of quick and instantaneous movements. 
Accordingly, a car moving through the landscape illustrated in Fig. 1 would not cause a macro 
displacement, but only a micro displacement of image infonnation which is ignored by the input 
device. 

[0025] Only macro displacements of the recorded image infomnation are taken into account for 
the inputting of control information or the control of selection infonnation P and P' displayed on 
the display unit 2, with a threshold decision element (not shown) setting an absolute magnitude 
for the macro displacement. Accordingly, a macro displacement is taken into account by the 
evaluation unit 3 if. for example, at least 90% of the image information recorded by the 
recording unit 1 exhibits the same or a similar displacement. A parallel displacement in 
essentially the same x and y direction is interpreted here ets the same displacement, while a 
displacement in the z direction is interpreted as a similar displacement. 

[0026] Fig. 2 shows a schematic diagram for illustrating a relative displacement of image 
infonnation in a z direction. According to Fig. 2, the image information recorded by the 
recording unit 1 comprises a house which is initially at a distance (image infonnation A) and the 
same house which, as a result of coming closer in the z direction, is displayed as enlarged by 
the image information A'. 

[0027] Accordingly, given a displacement in the z direction, the displacement vectors are not 
ordered parallel to one another in the x or y direction, but exhibit a concentric displacement 
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direction. Such concentrically oriented displacement vectors can therefore be evaluated by the 
evaluation unit 3 as control information in the z direction, thus enabling even three-dimensional 
control of the selection information displayed on the display unit 2, i.e. pointer symbols P and P. 
The display unit 2 may also be a three-dimensional display unit here (e.g. holographic display 
unit). 

[0028] As an altemative to moving the recording unit 1 in the z direction, i.e. moving closer to 
the object to be recorded, the recording unit 1 can also have an (electronic or mechanical) zoom 
function unit (not shown) with which it is likewise also possible to perfonn a concentric 
displacement of the image infonnation. The recording unit 1 may also have a rotating or 
swiveling unit (not shown) with which the macro displacements in the x and y directions can be 
realized. 

[0029] Figs. 3a and 3b show schematic diagrams of a mobile multimedia communication 
terminal having the input device according to a first exemplary embodiment of the invention. In 
Fig. 3a, MM denotes a mobile multimedia communication temiinal, as may be realized for 
example in the fomi of a mobile videophone with Intemet access. In addition to the usual 
function units, the mobile multimedia communication temiinal MM essentially has a recording 
unit 1, a display unit 2 and an activation button T for activating/deactivating the input device. In 
the case of an Internet access, the enlarged Intemet page IS (web page) is displayed on the 
display unit 2 for example, with a pointer symbol P being displayed as selection infonnation at 
the position L To move the pointer symbol P from position I to position II, according to Fig. 3b a 
user can, for example, move the mobile multimedia communication tenninal MM downward in 
front of his face, with the result that, instead of image information A at position I, the image 
information A' at position II is now recorded by the recording unit 1. The resulting macro 
displacements are detected by the evaluation unit 3. as described above, and are processed by 
the data processing unit 4 in such a way that the display unit 2 now displays the pointer P* at 
position II of the displayed Internet page IS. A displacement of the pointer symbol P showing 
the selection information to P' is thus achieved in this way by displacing the recording unit 1 
relative to the recorded background. 

[0030] As was already described above, it is also possible to perfomn a swiveling instead of 
the vertical displacement of the mobile multimedia communication terminal from a position I to a 
position II, which enables the pointer symbol P to be displaced in the same manner to position II 
for the pointer symbol P'. 
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[0031] Mobile multimedia communication terminals, as illustrated by way of example in Figs. 
3a and 3b, are especially advantageous if they are equipped as mobile videophones. Mobile 
videophones already have a large display unit 2 as well as a recording unit 1 for recording 
image information. Moreover, such mobile videophones already have an image compression 
device as image evaluation unit 3 for compressing the recorded image information. The 
algorithms used for image compression detect a movement in the Image infonnation A and A\ 
with the movement of the total image information being calculated in such a way to produce a 
total or main motion vector which is equivalent to the movement or displacement of the mobile 
multimedia communication terminal MM or the associated image infonnation A and A' 
respectively. This total or main motion vector, which can be readily calculated during image 
compression, can now be processed as control infonnation with corresponding x, y and z 
components, and displace the selection infonnation, i.e. the pointer symbols P or P', 
accordingly. Since the compression algorithms used are essentially already present in a mobile 
videophone, no additional costs are incun^ed. To switch over or activate/deactivate the input 
device, the mobile videophone or mobile multimedia communication tenninal MM requires only 
an activation button T. 

[0032] Fig. 4 shows a schematic diagram of a mobile multimedia communication tenninal 
having an input device according to a second exemplary embodiment. In Fig. 4 the same 
reference symbols denote the same or con^esponding components as in Figs. 3a and 3b, so a 
detailed description is dispensed with below. 

[0033] On a mobile multimedia communication terminal MM having an input device according to 
the second exemplary embodiment, the display unit 2 does not display the complete Internet 
page IS, but only a selected section F. This way of displaying of information, for example from 
the Internet or from another application, has the advantage^hat the information on the displayed 
page IS is more readily legible for the user. 

[0034] According to the second exemplary embodiment, the Input device now has intuitive 
control of the infonnation page IS to be displayed. To be more precise, a user need only 
displace the mobile multimedia communication terminal with the recording unit 1 in the x and y 
directions in order to reach the respective sections of the information page IS, so that the 
information page IS can be read intuitively (i.e. by displacing a viewing window) without using 
any further keys or input media. 
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[0035] In addition, a displacement in the z direction, i.e. a forward/backward movement of the 
mobile multimedia communication terminal MM, can enlarge or reduce the size of the selected 
section F, which enables intuitive use comparable to the use of a magnifying glass. 

[0036] According to Figs. 3 and 4, the recording unit 1 is located on the front of the mobile 
multimedia communication temiinal MM. It can however also be located on the rear of the 
temiinal, or a second recording unit may be arranged on the rear of the mobile multimedia 
communication temiinal MM. Given suitable realization of the optics, i.e. the ability to focus the 
recording unit 1 even at a very short distance, the mobile multimedia communication temiinal 
MM can also be placed rear down on a suitably structured surface and the whole unit can be 
displaced in the same way as a mouse. The recording unit located on the rear detects here, as 
described above, the relative movement or relative displacement of the recorded image 
information (structured surface) and determines from it the con^esponding control information for 
controlling the selection infomnation, i.e. a pointer symbol P or a display section F of the 
information page IS. This results in the input device being particularly easy to operate, since it 
essentially corresponds to a mouse input device. In this case the mobile multimedia 
communication terminal preferably has the selection buttons (not shown) known from mouse 
input devices. 

[0037] Figs. 5a to 5c show enlarged partial views of a mobile multimedia communication 
temiinal MM having an input device according to a third exemplary embodiment. In Figs. 5a to 
5c, the same reference symbols denote the same components as in Figs. 1 to 4, so a detailed 
description is dispensed with below. 

[0038] According to Figs. 5a to 5c, the input device according to the third exemplary 
embodiment can be used in the same way as a so-called "touch-pad" input device. In Fig. 5a, 
reference numeral 1 again denotes a recording unit 1, butSne which is a special macro camera 
system. To be more precise, the recording unit 1 according to Figs. 5a to 5c is able to record 
very fine structures in the immediate vicinity of the camera, by which, for example, the finger 
lines of a finger placed directly on the recording unit are recorded in sharp focus. 

[0039] According to Figs. 5a to 5c, the display unit 2 shows a telephone directory list with a 
plurality of names and corresponding phone numbers. The telephone directory list displayed on 
the display unit 2 thus corresponds to an infomnation page IS, with the selection information 
being represented neither by a pointer symbol P for indicating a selection position, nor by a 
display area F, but by a predetermined selection field B in the form of a. for example, colored 
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bar. To control the input, or to select the name Meier with phone number 345678, the user then 
places his finger on the recording unit 1 , with the input device activated by the activation button 
T (not shown). According to Fig. 5b, the finger lines are now recorded as image infomiation A, 
and the user moves his finger downward to displace the bar B. Fig. 5c shows here a partial 
view of the mobile multimedia communication terminal MM, with the finger shown at a lower 
position. The recording unit 1 accordingly detected a displacement of the recorded image 
information A ^ A' on the basis of the finger lines and evaluated this as a vertical displacement 
downward. The evaluation unit 3 accordingly outputs corresponding control information to the 
data processing unit 4, which results in the bar B being displaced downward on the display unit 
2 and now highlighting the desired name Meier with phone number 345678. 

[0040] To select the desired name, according to the third exemplary embodiment the input 
device can also have a pressure sensor for confirming the displayed selection information, i.e. 
the name highlighted by the bar B, which dispenses with the need for further fixed keys or so- 
called menu-driven "soft keys". 

[0041] Given suitable dimensioning of the recording unit 1 and the evaluation unit 3. in this way 
it is also possible to make an input in a way known from "touch pads" and/or "touch screens" for 
example. The integration of a pressure sensor in the recording unit 1 further reduces the space 
required for the input device and enhances ease of operation. In comparison with known "touch 
pads" in particular, with the input device according to the third exemplary embodiment, it is not 
the main focus of a surface area which is detennined, but rather the exact position or movement 
of the finger Is tracked on the basis of the lines present on it. The precision of this input device 
is therefore much greater than that of known input devices. With the input device according to 
the third exemplary embodiment, therefore, essentially a relative macro displacement of the 
recorded image Infomrjation is also used to control selection infomriation. 

[0042] A CCD or CMOS sensor camera is preferably used as the recording unit 1. The 
recording unit 1 is located here either on the front or on the rear of an input or computer system 
respectively, but it is not limited thereto and may also be an-anged at a different position and be 
moved around, for example by mirrors, in the same or opposite direction to the display direction 
of the display unit 2. 

[0043] The invention was described above with reference to a mobile multimedia 
communication terminal. It is however not limited thereto, and likewise encompasses ultra- 
small computer systems without a communication link, such as palmtop computers for example. 
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Likewise, the present invention may also be used on corded telecommunication terminals and 
other computer systems. 

[0044] The invention has been described in detail with particular reference to preferred 
embodiments thereof and examples, but it will be understood that variations and modifications 
can be effected within the spirit and scope of the invention. 



